Stereo telestration for robotic surgery

ABSTRACT

In one embodiment of the invention, a robotic surgical system includes a master control console having a stereo viewer to view stereo images; a surgical manipulator having a stereo endoscopic camera coupled to a robotic arm to generate the stereo images of a surgical site; a stereo telestration device coupled between the stereo endoscopic camera and the stereo viewer to mix telestration graphics and the stereo images of the surgical site together for viewing by the stereo viewer; and a telestration generator coupled to the stereo telestration device to generate the telestration graphics for overlay on the stereo images of the surgical site.

FIELD

The embodiments of the invention relate generally to telestrationsystems. More particularly, the embodiments of the invention relate totelestration mentoring systems for robotic surgery.

BACKGROUND

A telestrator is a device that allows its operator to draw a freehandsketch over a motion picture image. The act of drawing a freehand sketchover a motion picture image is often referred to as telestration. Thefreehand sketch may be referred to as a telestration image. Telestratorshave been used to annotate televised weather reports and televisedsporting events.

Telestration systems are often used in television broadcasts of footballgames to make a point to a television audience regarding one or moreplays during the game. A sports commentator may draw sketches ofobjects, such as X and O, circles or lines, that is overlaid anddisplayed on still or moving video images of the play on the televisionmonitor. Typically, the telestration image is displayed on a singletelevision monitor in a mono-visual (“mono-view”) format and viewed byboth eyes of the television viewer. The mono-view provided by the singletelevision monitor is limited to two dimensional images.

BRIEF SUMMARY

The embodiments of the invention are summarized by the claims thatfollow below.

BRIEF DESCRIPTIONS OF THE DRAWINGS

FIG. 1 is a block diagram of a robotic surgery system including a stereoviewer and a stereo telestration system to provide annotated stereoimages to a surgeon.

FIG. 2A is a block diagram of a first system to provide a stereotelestration image overlay in both left and right video channels toprovide three-dimensional images in a stereo viewer.

FIG. 2B is a block diagram of a second system to provide a stereotelestration image overlay in both left and right video channels toprovide three-dimensional images in a stereo viewer.

FIG. 3 is a perspective view of a robotic surgical master controlconsole including the stereo viewer.

FIG. 4 illustrates the stereo viewer of the master control console ofFIG. 3 with a stereo telestration image overlay in both left and rightmonitors to provide three-dimensional images of the surgical site andthe telestration images.

FIG. 5A illustrates a block diagram of a digital composite video mixerto mix a surgical site video signal and a telestration video signaltogether.

FIG. 5B illustrates a block diagram of a digital component video mixerto mix a surgical site video signal and a telestration video signaltogether.

FIG. 5C illustrates a block diagram of an analog video mixer to mix ananalog surgical site video signal and an analog telestration videosignal together.

FIG. 6A illustrates left and right annotated surgical site images.

FIG. 6B illustrates a three dimensional coordinate system for the stereoimages of a background object and the stereo telestration images.

FIG. 6C illustrates a stereo window of left and right images in thestereo viewer to show the horizontal offset between the lefttelestration image and the right telestration image to achieve fusingand the same depth.

FIG. 7 is a block diagram of an exemplary endoscopic camera.

FIG. 8 is a magnified perspective view of the exemplary endoscopiccamera and the plane of a tissue or objection.

FIGS. 9A-9C are diagrams to illustrate the generation of a disparitymap.

FIG. 10 is side perspective stereo view illustrating differences in atelestration mark generated at an apparent constant depth and atelestration mark generated with an apparent depth continuum to appearpainted onto a surface.

DETAILED DESCRIPTION

In the following detailed description of the embodiments of theinvention, numerous specific details are set forth in order to provide athorough understanding of the present invention. However, it will beobvious to one skilled in the art that the embodiments of the inventionmay be practiced without these specific details. In other instances wellknown methods, procedures, components, and circuits have not beendescribed in detail so as not to unnecessarily obscure aspects of theembodiments of the invention.

One application for telestration systems is robotic surgery. In roboticsurgery, two monitors are used to provide a stereo-visual(“stereo-view”) and a three-dimensional image to a pair of eyes. Thethree-dimensional image is important for depth perception of thesurgical site and viewing the robotic surgical tools perform surgery ona patient within the surgical site.

A mono-visual image in a single monitor to a single eye is lessdesirable in robotic surgery. Similarly, while a stereo image of thesurgical site is desirable, a mono-visual telestration image in only onemonitor of the pair of monitors is less desirable during roboticsurgery. With only a mono-visual telestration image, a surgeon may beconfused, as one eye sees one half of a stereo image without thetelestration image. Moreover, it may be hard on the surgeon's eyes andbrain to view a mono-visual telestration image for extended periods andcause fatigue during surgery which is undesirable.

A video frame or a frame of pixel data may be used interchangeably withimage herein. However, at a viewing device, an image is what isperceived by a user when viewing the video frame or pixel frame of dataon the viewing device. A stereo image with a pair of images (e.g., aleft image and a right image) has left and right video frames or leftand right frames of pixel data. A mono-visual image or mono-image hasone of a left image or a right image and one of a left or right videoframe or a left or right frame of pixel data.

The embodiments of the invention include a method, apparatus, and systemfor stereo telestration for robotic surgery.

Robotic Surgical System

Referring now to FIG. 1, a block diagram of a robotic surgery system 100is illustrated to perform minimally invasive robotic surgical proceduresusing a stereo telestration system. Robotic surgery generally involvesthe use of a robot manipulator that has multiple robotic manipulatorarms. One or more of the robotic manipulator arms often support asurgical tool which may be articulated (such as jaws, scissors,graspers, needle holders, micro dissectors, staple appliers, tackers,suction/irrigation tools, clip appliers, or the like) or non-articulated(such as cutting blades, cautery probes, irrigators, catheters, suctionorifices, or the like). At least one of the robotic manipulator arms 153(e.g., the center robotic manipulator arm 153) is used to support astereo or three dimensional surgical image capture device 110 such as astereo endoscope (which may be any of a variety of structures such as astereo laparoscope, arthroscope, hysteroscope, or the like), or,optionally, some other stereo imaging modality (such as ultrasound,fluoroscopy, magnetic resonance imaging, or the like). Robotic surgerymay be used to perform a wide variety of surgical procedures, includingbut not limited to open surgery, neurosurgical procedures (such asstereotaxy), endoscopic procedures (such as laparoscopy, arthroscopy,thoracoscopy), and the like.

A user or operator O (generally a surgeon) performs a minimally invasivesurgical procedure on patient P by manipulating input devices at amaster control console 150. A computer 151 of the console 150 directsmovement of robotically controlled endoscopic surgical instruments101A-101B and 110, by means of one or more feedback/control cables 159,effecting movement of the instruments using a robotic surgicalmanipulator 152. The robotic surgical manipulator 152 may also bereferred to as robotic patient-side cart system or simply as a cart. Therobotic surgical manipulator 152 has one or more robotic arms 153.Typically, the robotic surgical manipulator 152 includes at least threerobotic manipulator arms 153 supported by linkages 156,156′, with acentral arm 153 supporting an endoscopic camera 110 and the robotic arms153 to left and right of center supporting tissue manipulation tools101A-101B.

Generally, the robotic arms 153 of robotic surgical manipulator 152include a positioning portion and a driven portion. The positioningportion of the robotic surgical manipulator 152 remains in a fixedconfiguration during surgery while manipulating tissue. The drivenportion of the robotic surgical manipulator 152 is actively articulatedunder the direction of the operator O generating control signals at thesurgeon's console 150 during surgery. The actively driven portion of thearms 153 is herein referred to as an end effector 158. The positioningportion of the robotic arms 153 that are in a fixed configuration duringsurgery may be referred to as positioning linkage and/or “set-up joint”156, 156′.

An assistant A may assist in pre-positioning of the robotic surgicalmanipulator 152 relative to patient P as well as swapping tools orinstruments 101 for alternative tool structures, and the like, whileviewing the internal surgical site via an assistant's display 154.

The image of the internal surgical site shown to A by the assistant'sdisplay 154 is provided by a left or right channel 176 of the stereoendoscopic camera 110 supported by the robotic surgical manipulator 152.In contrast, both left and right channels of the stereo endoscopiccamera 110 are provided to the operator O in a stereo display 164 at thesurgeon's console 150, one channel for each eye.

Stereo Telestration

A teacher, instructor, or other person, referred to generally as mentorM, may be on site or at a remote location and use a telestrator togenerate telestration and provide comments and instructions to theoperator O regarding the robotic surgical procedure in the surgical siteof the patient P. In this manner an expert on the robotic surgicalprocedure, such as mentor M, may guide a less experienced operatingsurgeon O.

A typical telestration system provides mono-view images. The roboticsurgical system has a stereo viewer which displays a three dimensionalimage of the surgical site to the surgeon O. If the telestration imageis displayed in only one eye, confusion can result since the other eyeis seeing the other image of the stereo pair without the telestrationimage overlay. To support stereo telestration from the mentor M, therobotic surgical system 100 includes a stereo telestration system 160coupled between the console 150 and remote located telestrationequipment 161. The remote located telestration equipment 161 may belocated remotely in the same room as the patient and surgeon or in adifferent room, a different hospital, or a different city, country,continent or other differing location.

The stereo telestration system 160 processes left and right channels ofstereo video signals and optionally, full duplex audio signals foraudio/video communication. The stereo telestration system 160 receivesstereo images of the surgical site (“stereo surgical images”) from thestereo endoscopic camera 110 over the stereo video communication link170. A mono-view of telestration images (“mono telestration images”) isgenerated by a telestrator or telestration generator 162 (such as adrawing tablet 262 and drawing pen 263 illustrated in FIG. 2B forexample) and coupled into the stereo telestration system 160 over thecommunication link 172. The telestration generator 162 digitizes atelestration mark or telestration graphic into a digital telestrationgraphic image for communication over the link 172.

The stereo telestration system 160 overlays the mono telestration imagesonto the stereo surgical images of the surgical site generated by thestereo endoscopic camera 110 to form annotated stereo surgical images.The telestration system 160 couples the annotated stereo surgical imagesinto the stereo display 164 of the console 150 over the stereo videocommunication link 175 for viewing by the operator O. The telestrationsystem 160 may also couple the annotated stereo surgical images over avideo communication link 176 to a stereo viewer at a remote location forviewing by the person generating the telestration. Alternatively, asingle left or right channel of the annotated stereo surgical images maybe coupled by the telestration system 160 over the video communicationlink 176 to a single video monitor 165A at a remote location for viewingby the person generating the telestration.

While the telestration system 160 generates video images, it mayoptionally provide a full duplex audio communication channel between theoperator O and the person generating the telestration. Alternatively, awireless or wired telephone system, such as cellular telephone system,internet protocol telephone system, or plain old telephone system may beused to provide the full duplex audio communication channel.

The remote located telestration equipment 161 may include a videomonitor 165A, a telestration generator, a microphone 180, a speaker 181,and an audio processor 182 coupled together as shown. Telestrationimages are generated by the telestration generating device 162 that iscoupled to the stereo telestration system 160 over the communicationlink 172. A telestration generating device may also be referred toherein as a telestrator or a telestration generator.

In the case of a mono-view monitor, the video monitor 165A receiveseither the left or right channel of the annotated stereo surgical imagesover the video communication link 176 for viewing by the mentor M at theremote location. In the case of a stereo viewer for the mentor M, thestereo viewer receives both of the left and right channels of theannotated stereo surgical images over the video communication link 176at the remote location so that the mentor M may view stereo imagessimilar to the stereo viewer 164 in the console 150. That is, thecommunication link 176 may carry either one or both of a left or rightchannel of annotated surgical images.

As discussed previously, the stereo telestration system 160 may overlaya mono telestration image onto stereo images of the surgical site(referred to as “stereo surgical images”) generated by the stereoendoscopic camera 110 to form annotated stereo surgical images. Howeverin an alternate embodiment of the invention, the mono telestration imageis not immediately overlayed onto the stereo surgical images. Instead,the mentor M privately previews his telestration graphics overlayed ontothe surgical site images on the monitor 165A,165B before thetelestration graphics are overlayed onto the surgical site imagesdisplayed at the stereo viewer 164 to the operator O. That is, thementor M views the annotated surgical images before the telestrationgoes “live” on the stereo viewer for the operator O to see.

As previously discussed, the telestration system 160 may optionallyprovide a full duplex audio communication channel 184 between theoperator O and the mentor M. To support full duplex communication, theremote located telestration equipment 161 may include a microphone 180,a speaker 181, and an audio processor 182 coupled to the communicationchannel 184. The console may also include a microphone 180, a speaker181, and an audio processor 186 coupled to the channel 184 to supportfull duplex communication.

If cables cannot be used to reach the remote located telestrationequipment 161, modems, transceivers, or other communication devices191,192 may be used to form data/audio/video communication channels172,176,184 over a communication network 190. In one embodiment of theinvention, the communication network 190 is a wide area network such asthe internet and the communication devices 191,192 are wide area networkrouters. For the audio channel, hands-free telephones may be used ateach end to communication between remote locations over the plain oldtelephone system (POTS) of communication.

Referring now to FIG. 3, a perspective view of the robotic surgicalmaster control console 150 is illustrated. The master control console150 of the robotic surgical system 100 may include the computer 151, abinocular or stereo viewer 312, an arm support 314, a pair of controlinput wrists and control input arms in a workspace 316, foot pedals 318(including foot pedals 318A-318B), and a viewing sensor 320. The mastercontrol console 150 may further include the telestration system 160 forproviding the telestration images overlaid on the surgical site images.The master control console 150 may also include an audio processor ortransceiver 317 coupled to a speaker 320 and a microphone 315 for abi-directional voice communication system to provide full duplex voicecommunication between the operating surgeon O and the mentor M. Theaudio processor or transceiver 317 may couple to or be a part of thetelestration system 160 in embodiments of the invention.

The stereo viewer 312 has two displays where stereo three-dimensionalimages of the telestration and surgical site may be viewed to performminimally invasive surgery. When using the master control console, theoperator O typically sits in a chair, moves his or her head intoalignment with the stereo viewer 312 to view the three-dimensionalannotated images of the surgical site. To ensure that the operator isviewing the surgical site when controlling the robotic surgical tools101, the master control console 150 may include the viewing sensor 320disposed adjacent the binocular display 312. When the system operatoraligns his or her eyes with the binocular eye pieces of the display 312to view a stereoscopic image of the telestration and surgical worksite,the operator's head sets off the viewing sensor 320 to enable thecontrol of the robotic surgical tools 101. When the operator's head isremoved the area of the display 312, the viewing sensor 320 can disableor stop generating new control signals in response to movements of thetouch sensitive handles in order to hold the state of the roboticsurgical tools.

The arm support 314 can be used to rest the elbows or forearms of theoperator O (typically a surgeon) while gripping touch sensitive handlesof the control input wrists, one in each hand, in the workspace 316 togenerate control signals. The touch sensitive handles are positioned inthe workspace 316 disposed beyond the arm support 314 and below theviewer 312. This allows the touch sensitive handles to be moved easilyin the control space 316 in both position and orientation to generatecontrol signals. Additionally, the operator O can use his feet tocontrol the foot-pedals 318 to change the configuration of the surgicalsystem and generate additional control signals to control the roboticsurgical instruments.

The computer 151 may include one or microprocessors 302 to executeinstructions and a storage device 304 to store software with executableinstructions that may be used to generate control signals to control therobotic surgical system 100. The computer 151 with its microprocessors302 interprets movements and actuation of the touch sensitive handles(and other inputs from the operator O or other personnel) to generatecontrol signals to control the robotic surgical instruments 101 in thesurgical worksite. In one embodiment of the invention, the computer 151and the stereo viewer 312 map the surgical worksite into the controllerworkspace 316 so it feels and appears to the operator that the touchsensitive handles are working over the surgical worksite.

Referring now to FIG. 4, a perspective view of the stereo viewer 312 ofthe master control console 150 is illustrated. To provide athree-dimensional perspective, the viewer 312 includes stereo images foreach eye including a left image 400L and a right image 400R of thesurgical site including any robotic surgical tools 400 respectively in aleft viewfinder 401L and a right viewfinder 401R. The images 400L and400R in the viewfinders may be provided by a left display device 402Land a right display device 402R, respectively. The display devices402L,402R may optionally be pairs of cathode ray tube (CRT) monitors,liquid crystal displays (LCDs), or other type of image display devices(e.g., plasma, digital light projection, etc.). In the preferredembodiment of the invention, the images are provided in color by a pairof color display devices 402L,402R; such as color CRTs or color LCDs.

In the stereo viewer, three dimensional telestration images may beprovided to a surgeon by overlaying them onto the three dimensionalimage of the surgical site. In a right viewfinder 401R, a righttelestration image (RTI) 410R is merged into or overlaid on the rightimage 400R being displayed by the display device 402R. In a leftviewfinder 401L, a left telestration image (LTI) 410L is merged into oroverlaid on the left image 400L of the surgical site provided by thedisplay device 402L. In this manner, a stereo telestration image may bedisplayed to provide instructions to the operator O in the control ofthe robotic surgical tools in the surgical site.

Referring now to FIGS. 2A-2B, embodiments of stereo telestration imagingsystems are illustrated. In FIG. 2A, a first embodiment of the stereotelestration imaging system includes the stereo endoscopic camera 110,the telestration system 160, remote telestration equipment 161A, and thestereo viewer 164.

As discussed previously, the remote telestration equipment 161A includesa telestration generator 162A and a single video monitor 165A for thementor M to view a mono view of the annotated surgical site generated bythe telestration system 160. The remote telestration equipment 161A mayfurther include a part of a full duplex audio communication system suchas a telephone or speaker phone described previously with reference toFIG. 1.

The telestration generator 162A may include a drawing tablet 262 and adrawing pen 263, to generate the mono view telestration images foroverlay onto the stereo images of the surgical site. The drawing tablet262 and drawing pen 263 may also be referred to herein as a digitizingtablet and digitizing pen as they digitize a sketched drawing into adigital telestration graphic image. The telestration generator 162A mayalso include a keyboard 264. The telestration generator 162A mayadditionally or in the alternative include one or more elements of thetelestration generator 162B described in greater detail below.

As discussed previously for one embodiment of the invention, a mentor Mmay preview the telestration graphics that are to overlayed onto thesurgical site images on the monitor 165A,165B before the telestrationgraphics are overlayed onto the surgical site images displayed at thestereo viewer 164 to the operator O. Additionally, a mono-viewtelestration image may be generated for multiple video frames until anerase command is issued to the drawing tablet. That is, as the sketch ismade on the drawing tablet, the mono view telestration images show thegrowth of the sketch until completion, which is then shown in a steadystate until erased.

The stereo endoscopic camera 110 includes an endoscope 202 for insertioninto a patient, a camera head 204, a left image forming device (e.g., acharge coupled device (CCD)) 206L, a right image forming device 206R, aleft camera control unit (CCU) 208L, and a right camera control unit(CCU) 208R coupled together as shown. The stereo endoscopic camera 110generates a left video channel 211L and a right video channel 211R offrames of images of the surgical site. To initially synchronize left andright frames of data, a lock reference signal is coupled between theleft and right camera control units 208L,208R. In one embodiment of theinvention, the right camera control unit generates the lock signal thatis coupled to the left camera control unit to synchronize the left viewchannel to the right video channel. However in another embodiment of theinvention, the left camera control unit generates the lock referencesignal and the right video channel synchronizes to the left videochannel.

The stereo display 164 includes a left monitor 230L and a right monitor230R. As discussed previously with reference to FIG. 4, the viewfindersor monitors 230L,230R may be provided by a left display device 402L anda right display device 402R, respectively. In the preferred embodimentof the invention, the stereo images are provided in color by a pair ofcolor display devices 402L,402R.

Additional details of a stereo endoscopic camera and a stereo displaymay be found in U.S. Pat. No. 5,577,991 entitled “Three DimensionalVision Endoscope with Position Adjustment Means for Imaging Device andVisual Field Mask” filed on Jul. 7, 1995 by Akui et al; U.S. Pat. No.6,139,490 entitled “Stereoscopic Endoscope with Virtual Reality Viewing”filed on Nov. 10, 1997 by Breidenthal et al; and U.S. Pat. No. 6,720,988entitled “Stereo Imaging System and Method for use in TeleroboticSystems” filed on Aug. 20, 1999 by Gere et al.; all of which areincorporated herein by reference. Stereo images of a surgical site maybe captured by other types of endoscopic devices and cameras withdifferent structures. For example, a single optical channel may be usedwith a pair of spatially offset sensors to capture stereo images of thesurgical site.

The telestration device or system 160 for the left video channelincludes a left video combiner 210L and a left synchronizer/noisereducer/enhancer device 214L coupled to a VSD board 218; while the rightchannel includes a right video combiner 210R and a leftsynchronizer/noise reducer/enhancer device 214L coupled to the VSD board218. The telestration device or system 160 may further include left andright power transformers 240L-240R coupled to an isolation transformer242 to receive power.

The left video combiner 210L combines the telestration graphics orimages with the left video images of the surgical site on the left videochannel 211L. The right video combiner 210R combines the telestrationgraphics or images with the right video images of the surgical site onthe right video channel 211R. For the respective left and right videochannels, the left and right synchronizer/noise reducer/enhancer devices214L-214R perform analog-to digital conversion as necessary, pluselectronic noise reduction and image enhancement/sharpening in order toimprove (“sweeten”) the left and right images. Synchronization may alsobe provided by the devices 214L-214R however is not strictly necessarysince the camera control units (CCUS) are already synchronized. The VSDboard 218 performs interlaced-to-progressive video scan conversion;electronic image-shifting to correct endoscope and camera opticalmisalignment as is described further in U.S. Pat. No. 6,720,988 by Gereet al. (previously incorporated by reference); and control graphicoverlay for the respective left and right video channels.

The left and right video combiners 210L,210R may combine video signalsin various ways depending upon the type of video signals being provided.In one embodiment of the invention, the stereo video signals of thesurgical site provided on the left and right video channels 211L,211Rare analog video signals. In another embodiment of the invention, thestereo video signals of the surgical site provided on the left and rightvideo channels 211L,211R are digital video signals. Similarly, the monotelestration video signals on the link 172 are analog video signals inone embodiment of the invention and are digital video signals in anotherembodiment of the invention. Depending upon whether analog, digital, ormixed analog and digital video signals are used, various mixingtechniques may be employed to mix the stereo surgical site video signalswith the telestration video signals to form the stereo annotatedsurgical site video signals. Additionally, depending upon the format ofthe video signals (composite video or component video and theirrespective video formats e.g., RGB, S-Video or Y/C, YUV, YIQ, YCrCb),the type of mixing techniques used may vary to mix the stereo surgicalsite video signals and the telestration video signals together. In anycase, an alpha synchronizing signal may be provided that can be used tooverlay the graphic telestration images onto the video signal of thesurgical site.

Mixing two digital video sources may be simply performed by using amultiplexer to switch between sources or by soft keying by implementingfull alpha mixing. In FIG. 5A, two digital composite video signals eachhaving their own alpha channel are mixed together. The digital videosignal of the surgical site is coupled into the mixer 500A as one sourceand the digital video signal of the telestration image is coupled intothe mixer 500A as a second source. After subtracting out the digitalvalue of the black level at the subtractors 502A-502B, the sources arekeyed by their respective alpha signals alpha_0 and alpha_1 by thekeying device (e.g., multiplier) 504A-504B and then added together atthe summer or adder 506. The result from the summer 506 is then roundedand limited by a rounding/limiting device 508 to an appropriate numberof bits of digital video. The black level is then added back into thedigital video signal at the adder or summer 510 to generate theannotated surgical site video signal as the resultant output from themixer 500A.

For RGB component digital video signals, the mixing may be somewhatsimilar for each component signal. In FIG. 5B, an RGB component digitalvideo signal is provided for the surgical site video signal (SurgicalSite R_1, G_1, and B_1) and the telestration video signal (TelestrationR_1, G_1, and B_1) and coupled into the video mixer 500B. The resultantoutput from the video mixer 500B are the RGB components of the annotatedsurgical site video signal (Annotated Surgical Site R_out, G_out, andB_out). With the component video signals, the black level is typicallyzero by convention and therefore of little concern and this can besimplified from that of mixer 500A. For each component signal, thesources are keyed by their respective alpha signals alpha_0 and alpha_1by the keying devices (e.g., multipliers) 504A-504B to synchronize whenthe signals are to be added. The synchronized signals are then addedtogether at the summer or adders 506A-506C for each respective componentsignal. The result from each of the summers 506A-506C is then roundedand limited by the rounding/limiting devices 508A-508C to an appropriatenumber of bits of digital video to generate each respective RGBcomponent of the annotated surgical site video signal (AnnotatedSurgical Site R_out, G_out, and B_out).

FIG. 5C illustrates a simple analog video mixer 500C consisting of ananalog multiplexer 520 that is responsive to a keying signal coupled toits select terminal. The multiplexer 520 selects to output a videosignal from two input video signals. The multiplexer selects between thesurgical video signal coupled to one input terminal and the telestrationvideo signal coupled to a second input terminal.

In response to the keying signal, the multiplexer 520 can generate theannotated surgical site video signal. The keying signal is generated inresponse to a level of the input telestration video signal. In oneembodiment, the luminance level of the telestration video signal may beused as the keying signal. With the luminance of telestration videosignal above a predetermined level, the telestration video signal isselected to be output from the multiplexer 520. With the luminance levelof the telestration video signal below the predetermined level, thesurgical site video signal is selected to be output from the multiplexer520. In this manner, the Annotated surgical site video signal can begenerated by the mixer 500C.

If mixed analog and digital video signals are provided, the analog videosignal may be converted into a digital video signal and mixed accordingto digital mixing techniques. Alternatively, the digital video signalmay be used to key the analog video signal to select a monochrome imagein the analog mixing technique or the digital video signal may beconverted to an analog video signal and mixed according to analog mixingtechniques.

The right video combiner 210R may be a master video combiner feedingthrough the telestration graphics or images to a slave video combiner,the left video combiner 210L, over a communication link 272. In thiscase, the right video combiner 210R receives control/data signals andthe telestration images on the communication link 172 (at COMM IN input)from the remote telestration generator 162A. The COMM-OUT output of theright video combiner 210R is coupled to the COMM-IN input of the leftvideo combiner 210L by means of the communication link 272.Alternatively, the left video combiner may be the master combiner andthe right video combiner may be the slave combiner.

The remote telestration device 162A may couple to the telestrationsystem 160 through the communication link 172 over the communicationsystem 190 by means of the communication devices 191,192.

The telestration images on the communication link 172 are in a digitaldata format in a preferred embodiment of the invention. Thecommunication link 172 may use a standard RS-232 digital communicationprotocol as the telestration data may be simple X and Y coordinateswhich are not of high bandwidth.

As discussed previously, the right video combiner 210R may be coupled tothe left video combiner 210L by way of the communication link 272. Thecommunication link 272 may be another RS-232 link, for example. In thiscase, the right video combiner 210R simply relays the control/datasignals and the telestration images on the communication link 172 to theleft video combiner 210L over the communication link 272.

As discussed previously, the remote telestration equipment 161A includesthe single video monitor 165A for a mono view of the annotated surgicalsite generated by the telestration system 160. The video monitor 165Acouples to either a left annotated video channel 212L or a rightannotated video channel 212R of the annotated surgical images togenerate the mono view. The video monitor 165A may couple to either theleft annotated video channel 212L or the right annotated video channel212R over the communication system 190 by means of the communicationdevices 191,192.

Referring now to FIG. 2B, a second embodiment of the stereo telestrationimaging system is illustrated. The stereo telestration imaging systemincludes the stereo endoscopic camera 110, the telestration system 160,remote telestration equipment 161B, and the stereo viewer 164. Thestereo telestration imaging system of FIG. 2B, while substantiallysimilar to that of FIG. 2A, differs in the remote telestration equipment161B (e.g., includes a stereo viewer 165B instead of a monitor 165A) andhow it may be connected.

As previously discussed, the annotated stereo surgical images from thetelestration system 160 may be coupled over the video communication link176 to a stereo viewer 165B at a remote location for viewing by theperson generating the telestration, such as the mentor M. In this case,the stereo viewer 165B may couple to the left and right video channels220L,220R to receive the stereo annotated surgical images and displaythem in the left display L and the right display R for viewing by theleft and right eyes, respectively. Alternatively, the stereo viewer 165Bmay couple to the left and right video channels elsewhere in thetelestration system 160 after the telestration images are mixed with thesurgical site images, such as at left and right video channels 212L,212Rafter the devices 210L,210R or the left and right video channels216L,216R after the devices 214L,214R. In any case, the remote stereoviewer may couple to the telestration system 160 through the video link176 over the communication system 190 by means of the communicationdevices 191,192.

As mentioned previously, the remote telestration equipment 161 may beconnected differently. Instead of the left and right video combinersbeing connected to the telestrator device in a master-slaveconfiguration, they may be coupled in parallel to it. In this case, bothof the left and right video combiners 210L,210R receive control/datasignals and the telestration data signals over the communication link172 (at the COMM-IN inputs) from the remote telestration generator 162B.If for some reason analog video signals are used, the communication link172 may be split in two. If digital signals are used, the digital signalcan be readily fanned out into two signals as illustrated and coupledinto each communication input of the left and right video combiners210L,210R. The remote telestration generator 162B may couple to thetelestration system 160 through the communication link 172 over thecommunication system 190 by means of the communication devices 191,192.

The telestration generator 162B may include a computer 265, a keyboard264, and an input device 266 (such as a mouse, for example) to generatethe mono view telestration images for overlay onto the stereo images ofthe surgical site. The telestration generator 162B may additionally, orin the alternative, include one or more elements of the telestrationgenerator 162A, such as the drawing tablet 262 and the drawing pen 263described in greater detail above.

In yet another embodiment of the invention, the stereo telestrationimaging system of FIG. 2B is modified to include a three-dimensionalinput device 266 as part of the remote telestration equipment 161B withthe stereo viewer 165B. The three-dimensional input device 266 may be athree-D mouse or a duplicate of the three-D input control devices at themaster console 150. In this manner, a mentoring surgeon M could view athree dimensional surgical site and draw one or more telestration marksat a depth he/she desires by means of the three-dimensional input devicewithout need of any depth perception correction.

While FIGS. 2A-2B illustrate separate functional blocks for thetelestration device or system 160, such as the left video combiner 210Land the right video combiner 210R, a plurality of the functional blocksmay be incorporated into one integral electronic system, one integratedprinted circuit board, or one integrated circuit, such as the VSD board218 for example.

Depth Perception Correction for Stereo Telestration

In typical telestration systems, a telestration graphic image istypically placed in the foreground while the image being telestrated orsketched on is placed in the background. The telestration graphic imagemay be a pure opaque overlay so that background objects may be visible.This implies that the depth of the telestration graphic is no deeperthan the depth of the background object in order to preserve aforeground/background illusion.

In stereo telestration, the telestration image is displayed to both leftand right eyes as is discussed above.

By simply mixing the stereo surgical site with a mono-view telestrationimage, there may be a perceived difference in depth between the surgicalsite image and the telestration image in the annotated stereo surgicalsite image. Moreover, the left and right telestration images derivedfrom the mono view of the telestration image may not fuse into a stereoor three dimensional image. In some cases, this may not matter and nodepth perception correction is needed. However if a mono-viewtelestration image is used to generate stereo telestration, it isdesirable to correct for the differences in depth perception between thesurgical site image and the telestration image in most applications.That is, it is desirable to fuse the left and right telestration imagestogether in the stereo viewer at the same apparent depth of the surgicalsite stereo image when using a mono-view telestration image.

Note that typically the telestration images are placed at a depth lessthan or equal to the surgical site image and not greater, if thesurgical site image is the background. Placing the telestration imagesat a depth equal to the dept of the surgical site image is particularlyuseful when a mono view telestration image is generated by the mentorfrom a mono view. However, if the mentor has a stereo view and candirectly generate a stereo image of the telestration graphics, placingthe telestration images at a depth equal to the depth of the surgicalsite is less important. In which case, stereo image of the telestrationgraphics can be placed at a depth less than the depth of the surgicalimage because both mentor and operator viewing stereo telestrationimages can agree on the interpretation of the telestration graphic.

Referring now to FIG. 6A, a left image 602L and a right image 602R of anannotated stereo surgical site image is illustrated. The right image602R includes a right telestration image 610R in the surgical sitearound the needle 605. Simply mixing the mono telestration image drawnwith respect to the right channel may result in a left telestrationimage 610L offset within the surgical site from the needle 605 asillustrated in FIG. 6A. In this case, the telestration graphic ispositioned at a depth other than the foreground depth and it cannotuniquely identify any particular point to an operator O.

Referring now to FIG. 6B, it is desirable to adjust the perceived depthof the stereo telestration image 612A or 612B to the perceived depth ofthe object of interest 611. The telestration image is adjusted to thesame depth of the background object so that the stereo telestrationimage may uniquely identify a background location. In one case, thehorizontal position of one half of the stereo pair of images is adjustedfurther away from the other so as to move the stereo telestration image612A down towards the perceived depth of the object of interest 611. Inanother case, the horizontal position of one half of the stereo pair ofimages is adjusted closer to the other so as to move the stereotelestration image 612A up above the perceived depth of the object ofinterest 611.

Referring now to FIG. 6C, a stereo window 620 of the annotated stereosurgical site is illustrated having a left image 621L and a right image621R that may be viewed in the stereo viewer. The images in the stereowindow may be moved in depth with respect to the plane of the stereowindow by adjusting the stereo base or horizontal offset of the images.

Assuming the right channel was used by the mentor to generate a righttelestration image 612R around the right image 611R of the object ofinterest, the left telestration image 612L1 or 612L2 is horizontallyadjusted to fuse and form a stereo telestration image at the perceiveddepth of the stereo image 611 of the object of interest. The horizontalseparation distance 625 between the left telestration image 612L1 or612R and the right telestration image 612R may also be referred toherein as the horizontal offset or stereo base.

To move the stereo telestration image 612A down towards the perceiveddepth of the object of interest 611, the horizontal position of the lefttelestration image 612L2 in the left image 621L is adjusted further awayfrom the right telestration image 612R to a position of the lefttelestration image 612L1, for example, to fuse and form the stereotelestration image at the perceived depth of the stereo image 611 of theobject of interest. That is, the horizontal separation or horizontaloffset is increased. Alternatively, to move the stereo telestrationimage 612B up towards the perceived depth of the object of interest 611,the horizontal position of the left telestration image 612L1 in the leftimage 621L is adjusted closer to the right telestration image 612R to aposition of the left telestration image 612L2, for example, to fuse andform the stereo telestration image at the perceived depth of the stereoimage 611 of the object of interest. That is, the horizontal separationor horizontal offset is decreased.

In an alternate embodiment of the invention, the left or right image ofthe surgical site associated with the non-view channel is adjustedhorizontally to move the perceived depth of the surgical image deeper inthe stereo window or shallower in the stereo window. In yet anotherembodiment of the invention, the left and right telestration images areboth adjusted horizontally to move close together or farther apart so asto adjust the perceived depth in the stereo window. In yet anotherembodiment of the invention, the left and right surgical site images areboth adjusted horizontally to move close together or farther apart so asto adjust the perceived depth in the stereo window. Moving the left andright images further apart in the stereo window, increasing thehorizontal offset, moves the stereo image farther away, increasing theperceived depth of the stereo image. Moving the left and right imagescloser together in the stereo window, decreasing the horizontal offset,moves the stereo image closer, reducing the perceived depth of thestereo image.

In the case of a mono-view being provided to the mentor, for theoperator O to view a telestration image on the stereo viewer so that itis fusible with the left and right images of the surgical site, thetelestration image associated with the video channel not viewed by thementor is positionally adjusted. For example, in FIG. 2A the rightchannel 212R of the annotated surgical site video signal is viewed bythe mentor M over the video monitor 165A. The mentor generates thetelestration graphic images relative to the right video channel 211Rimages of the surgical site video signal so that it appears at thecorrect position therein.

The left video channel 211L images of the surgical site video signal maynot viewed by the Mentor M and may be referred to as the “non-viewedchannel”. In which case, the position of the telestration imageassociated with the non-viewed channel, left video channel 211L of thesurgical site, is positionally adjusted. For example, in FIG. 6 theposition of the left telestration image 610L is adjusted to correct forthe offset so that it is similarly positioned around the needle 605 asillustrated in the right image 602R.

The telestration images for the non-viewed channel are positionally(i.e., horizontally assuming parallel camera and viewer/eyes) adjustedso that telestration images and the surgical site images are fusible andappear at the same depth, as located by the mentor. The telestrationimages for the non-viewed channel may be automatically adjusted inposition by the stereo telestration video system or it may be manuallyperformed.

For the surgeon O to adjust the horizontal offset of the left and rightimages, the robotic surgery system 100 may further include a controlinput 187, such as a control knob, at the console 150. The control inputmay generate one or more control signals onto one or more control lines186 to control the stereo telestration system 160. Alternately, thecontrol input may mechanically or electromechanically control the stereoendoscopic camera 110 through one or more control lines 159.

For manual adjustment, a manual control input such as a control knob inthe console 150 may be provided to allow the surgeon O in someembodiments of the invention to adjust the horizontal position of atleast one of the left or right telestration images until they arefusible together.

The control knob may be used to generate an electronic control signal tocontrol the mixing of the telestration image with the surgical siteimage for one channel. In this case, the electronic control signal mayalter the alpha signal in a digital mixer or the keying signal in ananalog mixer, as to where the telestration image is to be overlaid ontothe surgical site image. Alternatively, the electronic control signalmay cause a horizontal shift in the position of the digital pixel dataof the telestration image in the video signal on one channel withrespect to the surgical site image. In some embodiments of theinvention, the control knob may be used to mechanically orelectro-mechanically (e.g., by electric motor control) control the leftor right channels of the endoscopic camera 110 to move a left or rightimage of the surgical site to be properly located under the telestrationimage.

In other embodiments of the invention, the robotic surgery system 100may further include a control input 187′, such as a control knob, thatmay be manipulated by the mentor M at the remote telestration equipment161 to generate an electronic control signal transmitted to thetelestration system 160. The control input 187′ may generate one or morecontrol signals onto one or more control lines 186′ to control thestereo telestration system 160 as further described herein. Alternately,the control input may mechanically or electromechanically control thestereo endoscopic camera 110 as further described herein through the oneor more control lines 186′. If local cabling is unavailable, the controlsignals for the one or more control lines 186′ may be communicated overthe communication link 190 by means of the communication devices191,192.

Referring now to FIG. 7, a block diagram of an exemplary endoscopiccamera 110 is illustrated. The exemplary endoscopic camera 110 includesa left observation optical system 702L and a right observation opticalsystem 702R in the endoscope 202. The exemplary endoscopic camera 110further includes a first mirror 711L and a second mirror 712L and one ormore image formation lenses 703L-704L in the left channel and a firstmirror 711R and a second mirror 712R and one or more image formationlenses 703R-704R in the right channel as part of the camera head 204.

The exemplary endoscopic camera 110 further includes a focusingarrangement. The lenses 704L and 704R may be adjusted in position by aposition adjustment mechanism 724 to focus left and right images intothe left and right cameras 206L,206R, respectively. The positionadjustment mechanism 724 may be moved by an electric motor 784 throughan appropriate transmission coupled there-between. A position sensor 785may be coupled to the position adjustment mechanism 724, the motor 784or the transmission coupled there-between to obtain a measure of focusposition. The motor 784 is controlled by means of a focus controller 786that is typically connected to an input device at the console.

The left and right cameras 206L,206R couple to the camera head 204 toreceive the respective left and right images of the surgical site toprovide a stereo image thereof. The cameras 206L,206R in one embodimentof the invention are charge coupled devices to generate a digital videosignal. The exemplary endoscopic camera 110 further includes the leftand right camera control units 208L,208R coupled to the left and rightcameras 206L,206R.

In one embodiment of the invention, the left and right cameras 206L,206Rare movable about the respective optical axes 750L,750R of the camerahead 204 by position adjustment mechanisms 706L,705R. That is, theposition adjustment mechanisms 706L,705R adjust the relative positionsof the cameras 206L,206R with respect to the left and right opticalsystems. In this manner, the position adjustment mechanisms 706L,705Rcan be used to manually adjust the horizontal position of the left orright cameras 206L-206R by a control knob 187,187′ to move a left orright image of the surgical site so that it is properly located underthe telestration image.

In another embodiment of the invention, the mirror 712L and the and oneor more image formation lenses 703L-704L in the left channel are movableby a position adjustment mechanism 714L while the mirror 712R and theone or more image formation lenses 703R-704R in the right channel aremovable by a position adjustment mechanism 714R. In this manner, theposition adjustment mechanisms 714L,714R may move the left or rightoptical axes 750L,750R of the camera head 204 under the left and rightcameras 206L,206R by a control knob 187,187′ to move a left or rightimage of the surgical site so that it is properly located under thetelestration image.

As discussed previously, the control knob for adjusting the position ofthe left or right telestration image may also be manipulated by thementor M at the remote telestration equipment instead of the operator Oat the console. The control knob 187′ of the remote telestrationequipment under control of the mentor M generates an electronic controlsignal transmitted to the telestration system 160 over communicationlink 190 through the communication devices 191-192. In this case, thementor M views both left and right channels of the stereo pair of imagessuch as illustrated in FIG. 2B. This allows the mentoring surgeon M toview the same stereo pair of images as the operating surgeon O.

Closing one eye (or using some functionally similar technology such as ashutter on the left or right video image), the mentoring surgeon M“marks” one half or side (i.e., one of the left or right channel) of thestereo pair with a telestration marking instrument. The telestrationsystem duplicates the mark in the other half or side of the stereo pairdisplays both to the operating surgeon O and mentor M. The mentoringsurgeon M then uses the control knob 187′ of the remote telestrationequipment 161 to adjust the horizontal offset of the second mark (withrespect to the first mark) until the stereo representation of the markappears to be at the correct depth with respect to whatever thementoring surgeon M determines is appropriate.

The control knob 187,187′ may be a generic control input device, whichcould be replaced with some other input device capable of representing acontinuum of choices in the horizontal offset of the telestration image.

The automatic positional adjustment of the telestration image in thenon-viewed channel uses a plurality of values for the position of theendoscopic camera in relationship to the surgical site, such as aplurality of distances between the endoscopic camera and the tissue at aplurality of points of the surgical site and a plurality of anglesbetween lines from the endoscopic camera to the points in the tissue andline segments between the respective points.

FIG. 8 illustrates a first distance 801A between the end of theendoscopic camera 110 and a first point P1 on a plane of tissue 800. Thefirst distance 801A represents the depth of the object of interest inthe stereo field of view at the first point P1, with P1 being in thetissue plane and along the centerline C of the endoscopic camera 110.FIG. 8 further illustrates a second point P2 and a third point P3 on thetissue 800 with respective distances 801B-801C between a line of sightof the range finder 805 and the plane of the tissue 800. Additionallyone may define angles 802A-802D representing the angles between thevarious line-of-sight line segments 801A-801C and the line segmentsbetween the points P1-P3 as illustrated.

As previously discussed, a plurality of points P on the tissue 800 withrespective angles 802 and distances 801 may be used to determine thehorizontal offset. If one angle 802A,802D between the camera and thetissue is known, at least two distances (801A,801B or 801A,801C) betweenat least three points (P1,P2, and P3) may be used to determine theorientation of the tissue plane 800 and hence the horizontal offset atany point on that plane. Otherwise, at least three distances(801A,801B,801C) between the camera and the tissue to at least threepoints (P1,P2,P3) may be used to determine the horizontal offset.

Several sensing or computing modalities may be used to determine orestimate the distance 801 that represents the depth of the object ofinterest in the stereo field. The sensing techniques may use hardware,software, or a combination thereof.

In one embodiment of the invention, one or more range finders 805similar to that used in auto-focus cameras may be used to determine thedistances 801A-801C. In another embodiment of the invention, thedistances 801A-801C may be computed from the position sensed by thefocus sensor 785 associated with the focus motor 784 of the focusingarrangement of the endoscopic camera.

The one or more angles 802A-802C between the endoscopic camera 110 andthe respective one or more points P1-P3 on the tissue plane 800 may bedetermined by using a plurality of range finders 805. Alternatively, theone or more angles may be determined by using a scanning range finderthat scans in a circle around an axis on the tissue plane 800. Without arange finder, angles may be determined using known tool tip locations inthe surgical site acquired during an initialization sequence, forexample. Such an initialization sequence may ask the operator O toprovide the location of the tissue plane to the electronics system bytouching it with the system's surgical instruments, which may bepositionally encoded to supply joint angles. As is appreciated by thosein the art, one may deduce the position of the instrument tips relativeto the endoscopic camera tip if all joints are encoded and thekinematics are known.

In yet another embodiment of the invention, image processing is used inthat left and right images of the tissue in a surgical site are capturedor registered as digital pixels into respective left and right digitalarrays similar to the one array illustrated in FIG. 13 of U.S. Pat. No.6,720,988. A three dimensional model of the left and right images arefurther formed similar to that described and illustrated in FIGS. 15 and16 of U.S. Pat. No. 6,720,988. The depth of the central feature in thethree-dimensional model at point 128 in FIG. 16 of U.S. Pat. No.6,720,988 may be used to represent the distance 801A, for example.

Other image processing methods may be used to compare the left and rightimages of the tissue in a surgical site to determine a measure for thedistance 801, such as spatial correlation, where the spatial delayprovides an indication of the desired horizontal offset (“the crucialnumber”) between the left and right telestration images to fuse themtogether at an appropriate depth.

In another embodiment of the invention, a depth map may be generated bysoftware to judge the depth of a surgical site and render thetelestration images at that depth. A depth map may be constructed byseveral ways known in the field of computer vision depth estimationincluding generating a depth map from the stereo images of the surgicalsite using left and right image correlation. Alternately, a depth mapcould be generated by a scanning range sensor, or similar raster depthmeasurement instrument, attached or otherwise registered to theendoscope tip.

In yet another embodiment of the invention, a disparity map may be usedto indicate how a pixel in the left eye should be associated with apixel in the right eye. In a number of computer vision depth estimationalgorithms, a depth map is formed by first creating a disparity map.With a disparity map, a depth map need not be created as the disparitymap may be used directly to generate a stereo telestration graphic atdesired depths. In some cases, a disparity map is created from a puredepth map (such as from a scanning range finder for example) to generatethe stereo telestration mark.

Referring now to FIGS. 9A-9C, ignoring well known issues of occlusionfor the purpose of simplification, diagrams illustrating the generationof a disparity map are now described. In FIG. 9A, the endoscopic camera110 scans the surgical site 900 within its field of view using the itsleft and right image forming devices 206L,206R. A feature A 902 in thesurgical site 900 is received and scanned by different areas and pixelsof the left and right image forming devices 206L,206R.

FIG. 9B illustrates left pixels of an exemplary left image 906L andright pixels of an exemplary right image 906R in the field of view ofsurgical site including the feature A 902 scanned in FIG. 9A. Theexemplary left image 906L includes a matrix of a plurality of leftpixels LP0,0 through LPM,N on N left scan lines 910L. The exemplaryright image 906R includes a matrix of a plurality of right pixels RP0,0through RPM,N on N right scan lines 910R.

The feature A 902 scans into the left and right images 906L,906R atdifferent horizontal pixel locations along respective scan lines 910L-Aand 910R-A. From an edge (e.g., the left edge) of the left image, a lefthorizontal distance d₁ along the scan line 910L-A can be determined tothe scanned location of the feature A 902. From a similar edge (e.g.,the left edge) of the right image, a right horizontal distance d_(r)along the scan line 910R-A can be determined to the scanned location ofthe feature A 902.

Ignoring issues of occlusion, the disparity DP of the feature A 902between right and left images may be determined by the equationDP=d_(r)−d_(l). Similarly, a disparity DP_(x,y) for each pixel alongscan lines in the right image 906R may be determined in comparison withpixels in corresponding scan lines in the left image 906L to form adisparity map. Alternatively, a disparity DP_(x,y) for each pixel alongscan lines in the left image 906L may be determined in comparison withpixels in corresponding scan lines in the right image 906R to form adisparity map. Typically a mixture of feature-based matching andinterpolation is employed to provide a DP_(x,y) for every single pointin one image, relative to the other image, where interpolation is usefulto match points with which no feature is clearly associated.

Referring now to FIG. 9C, a matrix 950 of disparities DP_(x,y) for eachpixel in one image (right or left) forms the disparity map between rightand left images. DP0,0 represents the disparity for one of the leftpixel LPX0,0 or right pixel RPX0,0. Similarly, DPm,n represents thedisparity for one of the left pixel LPXm,n or right pixel RPXm,n.Assuming that the left image is the base image, the disparity map forthe right image and its pixels RPX0,0 through RPXm,n is to be determinedsuch as by the equation DP_(x,y)=drRPX_(x,y)−dlLPX_(x,y).

A depth map is related to the disparity map by elementary geometricrelationships. Given the optics of the viewer and/or endoscope, a depthmap can be deduced from the disparity map. With the depth map and pixelsof the left image 906L as the base image, most of the right image 906Rmay be generated, but for right-eye scenes that are occluded in the lefteye.

While the horizontal offset between the left and right telestrationimages may be used to set a depth of the stereo telestration image, atwo-dimensional (“depth-less”) telestration mark or image may be“painted” onto a surgical site over a continuum of depths. That is, atelestration mark, drawing, or image may be drawn on top of one (e.g.,the left) image of the stereo pair, and the artificial disparity in theother image (e.g., the right) of the stereo pair is created at a varietyof depths, including different depths for different parts of thetelestration mark. Digital image processing techniques may be applied togenerate a continuum of depths for the stereo telestration image.

Referring now FIG. 10, a side perspective view of a surgical site toillustrate differences between a telestration mark having an apparentconstant depth and a telestration mark having a depth continuumgenerated by a disparity map, such as the disparity map matrix 950,between left and right images of the surgical site.

A surface 1000 of tissue for example in a surgical site is captured by acamera from above the tissue and viewed in stereo by a stereo viewer.The surface 1000 is uneven having varying surface characteristics thatare viewed at differing depths in the field of vision of the stereoviewer.

A mentor M generates a mono-view of a telestration mark 1002A using atwo dimensional input device. The mono view telestration is transformedinto a stereo view of left and right telestration images that are fusedtogether and overlayed over the surface 1000 in the surgical site usinga single horizontal offset value. Alternatively, the mentor M maygenerate a stereo view of the telestration mark using a threedimensional input device but it is constrained to be above the surface1000. In either case, the telestration mark 1002A may appear to behovering at an apparent constant depth over the varying surface 1000.

Instead of generating the telestration mark 1002A at a constant depth, a“painted” telestration mark 1002B may be generated that appears to bepainted onto the varying surface 1000 over its depth continuum. Theconstant depth telestration mark 1002A may be generated using a singlehorizontal offset value and a mono-view telestration image as previouslydiscussed with reference to FIGS. 6B-6C. In contrast, the “painted”telestration mark 1002B may be generated using the pixels of themono-view telestration image and a disparity map with disparities foreach pixel.

For example, assume the mono-view of the telestration image is directlycoupled to the left image for viewing by a left eye of the operator. Thedisparity map is applied to the pixels of the left image to transformthem into pixels for the right image. The transformed pixels of theright image are viewed by the right eye of the operator. As thedisparity map was generated using each pixel, the right image can begenerated on a pixel-by-pixel basis so that when viewed by a stereoviewer, the mark 1002B appears to be painted on top of the surface 1000.

Visual feedback may be provided to show the difference between theplacements of the constant depth telestration mark 1002A and the paintedtelestration mark 1002B. For example, the constant depth telestrationmark 1002A may be viewed as a red color image in the stereo viewer andthe “painted” telestration mark 1002B may be viewed as a blue colorimage on the surface 1000 in the stereo viewer.

As discussed previously, the horizontal offset between the left andright telestration images may be a function of one or more distances801A-801C and one or more angles 802A-802C. Regardless of how thedistances and angles are determined, it is desirable to determine theamount of horizontal offset between the left and right telestrationimages to represent a point in space as points in a stereo pair, suchthat the left and right telestration images fuse together and theoperator O perceives the point as being at the appropriate depth, whichin some cases is at the same apparent depth as the object of interest inthe stereo pair image. It is advantageous to adjust the position of thetelestration image so that the operator O can view a three-dimensionalimage on a stereo viewer with a telestration overlay, without beingconfused or distracted by a non-fused stereo telestration image.

While certain exemplary embodiments of the invention have been describedand shown in the accompanying drawings, it is to be understood that suchembodiments are merely illustrative of and not restrictive on the broadinvention, and that the embodiments of the invention not be limited tothe specific constructions and arrangements shown and described, sincevarious other modifications may occur to those ordinarily skilled in theart. For example, elements of one embodiment of the invention may beswapped for or combined with elements of another embodiment of theinvention. As a further example, the control knob 187,187′ to controlthe position of a left or right telestration image may be one or more ofcontrol buttons, keys, wheels, track ball, or other control inputdevice. Rather, the embodiments of the invention should be construedaccording to the claims that follow below.

1-16. (canceled)
 17. A method for a robotic surgical system comprising:receiving a mono-view of a telestration graphic; generating astereo-view of the telestration graphic from the mono-view of thetelestration graphic; and overlaying the stereo-view of the telestrationgraphic onto a stereo-view of a surgical site.
 18. (canceled)
 19. Themethod of claim 17, wherein the generating of the stereo-view of thetelestration graphic includes determining a disparity map between pixelsin left images and right images of the stereo view of the surgical siteto position the stereo view of the telestration graphic at one or moredepths in the stereo-view of the surgical site.
 20. The method of claim17, wherein the generating of the stereo-view of the telestrationgraphic includes determining a depth map between left images and rightimages of the stereo view of the surgical site to position the stereoview of the telestration graphic over a continuum of depths in thestereo-view of the surgical site.
 21. The method of claim 17, whereinthe overlaying of the stereo-view of the telestration graphic onto thestereo-view of the surgical site includes combining a left image of thetelestration graphic with a left image of the surgical site, andcombining a right image of the telestration graphic with a right imageof the surgical site.
 22. The method of claim 21, wherein the combiningof the left and right images of the telestration graphic with the leftand right images of the surgical site are by mixing in response to asignal. 23-32. (canceled)
 33. A method for telestrating on athree-dimensional image of a surgical site, comprising: receiving atelestration graphic input associated with one of a pair of stereoscopicimages of an surgical site; determining a corresponding telestrationgraphic input in the other of the pair of stereoscopic images using atleast one of a disparity map and a depth map corresponding to the pairof stereoscopic images so that a three-dimensional view of thetelestration graphic input is generated which is displayable as acontoured overlay to a three-dimensional view of the surgical site; anddisplaying the three-dimensional view of the telestration graphic inputas the contoured overlay to the three-dimensional view of the surgicalsite on a three-dimensional display.
 34. The method according to claim33, further comprising: transmitting information for the one of the pairof stereoscopic images to a location prior to receiving the telestrationgraphic input from the location.
 35. The method according to claim 34,wherein the location is a computer operated by a mentor.
 36. The methodaccording to claim 34, further comprising: receiving information for thepair of stereoscopic images prior to transmitting the information forthe one of the pair of stereoscopic images to the location.
 37. Themethod according to claim 36, wherein the information for the pair ofstereoscopic images is received from a stereoscopic endoscope.
 38. Themethod according to claim 37, wherein the pair of stereoscopic imagescomprises corresponding right and left camera views.
 39. The methodaccording to claim 33, further comprising: displaying thethree-dimensional view of the telestration graphic input as anon-destructive graphics overlay to the three-dimensional view of thesurgical site.
 40. A medical robotic system providing three-dimensionaltelestration comprising: a stereo endoscopic camera; a console having athree-dimensional display for displaying stereo images captured by thestereo endoscopic camera; a telestration generator receiving at leastone of a pair of stereo images captured by the stereo endoscopic camerato generate a two-dimensional telestration graphic input relative to thereceived stereo image; and a stereo telestration system configured toreceive the two-dimensional telestration graphic input from thetelestration generator, determine a corresponding two-dimensionaltelestration graphic input in the other of the pair of stereoscopicimages using at least one of a disparity map and a depth mapcorresponding to the pair of stereoscopic images so that athree-dimensional view of the telestration graphic input is generatedwhich is displayable as a contoured overlay to a three-dimensional viewof the surgical site on the three-dimensional display.
 41. The medicalrobotic system according to claim 40, wherein the telestration generatorincludes one or more of a digitizing tablet coupled to the stereotelestration system, a digitizing pen coupled to the digitizing tablet,a first keyboard coupled to the digitizing tablet, a computer coupled tothe stereo telestration system, a three-dimensional input device coupledto the computer, a mouse coupled to the computer, and a second keyboardcoupled to the computer.
 42. The medical robotic system according toclaim 41, wherein the two-dimensional telestration graphic input isgenerated from input provided by a mentor operating the telestrationgenerator.